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ABSTRACT 



The Teacher Professional Development Institute employs 
Internet technology to support distributed professional learning communities. 
The Secondary Teacher Education Program (STEP) at the University of 
Wisconsin-Madison will soon employ a similar model in its program. Work and 
learning within STEP evolves around instructional design teams comprising 
pre-service teachers and various advisors. Because much of this work is 
project -based rather than curriculum driven, traditional assessment is not 
appropriate. Our approach involves taking advantage of the online 
environment's ability to collect data on collaborative process. To help 
determine standards of evidence appropriate for judging groups as 
knowledge-construction entities, we articulated a theoretical framework based 
on four accepted views of social knowledge construction. Assessment was 
conceptualized as abductive inference, case identification given particular 
theories and relevant evidence pertaining to them. The theories predict 
indicators that can be represented as nodes linked by subjective 
probabilities within a Bayesian network. Observable features of online 
interactions provide input to the Net. Outputs are probability distributions 
indicating the degree to which a group meets normative standards described by 
theory. The network can produce general assessment and diagnostic profiles 
for groups and subgroups within large communities. (Author/SAH) 
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Abstract 




The Teacher Professional Development Institute employs Internet technology to support distributed professional 
learning communities. The Secondary Teacher Education Program (STEP) at the University of Wisconsin-Madison 
will soon employ a similar model in its program. Work and learning within STEP evolves around instructional design 
teams comprising pre-service teachers and various advisors. Because much of this work is project-based rather 
than curriculum driven, traditional assessment is not appropriate. Our approach involves taking advantage of the 
on-line environment's ability to collect data on collaborative process. To help determine standards of evidence 
appropriate forjudging groups as knowledge-construction entities, we articulated a theoretical framework based 
on four accepted views of social knowledge construction. Assessment was conceptualized as abductive inference, 
case identification given particular theories and relevant evidence pertaining to them. The theories predict 
indicators that can be represented as nodes linked by subjective probabilities within a Bayesian network. 
Observable features of on-line interactions provide input to the net. Outputs are probability distributions indicating 
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• the degree to which a group meets normative standards prescribed by theory. The network can produce general 
assessments and diagnostic profiles for groups and subgroups within large communities. 



Introduction 

New educational approaches are emertging that employ Internet technology to support collaborative environments 
for distributed professional learning communities. One example is the Teacher Professional Development Institute 
(TAPPED IN, or "TI" for short), although other examples have recently emerged in a variety of professional fields. 
When users log into the TI environment on the web (http://www.tappedin.org) they are located, virtually, in an 
institute. The institute is based on a building metaphor with floors, offices, meeting rooms, and other spaces that a 
physical institute would have. The virtual spaces of TI provide places where teachers can "meet" and where 
artifacts can be created and stored. The existing TI membership of over 2000 educators provides a broad national 
community basis for teacher professional development through collaborative learning fll. 

Designing an Instructional Program within an On-Line Environment 

Inspired by the TI example, our research group is helping restructure the Secondary Teacher Education Program 
(STEP) at the University of Wisconsin-Madison in accordance with a distributed professional learning community 
model. Briefly, work and learning within STEP evolves around instructional design projects that students develop to 
partially fulfill their course requirements. Each student in the program participates on one or more teams. Students 
direct the teams but are advised by various mentors who share in teamwork. Mentor-participants include 
in-service teachers in whose classrooms the STEP instruction will be evaluated; university faculty from the School 
of Education; and experts in subject-matter disciplines that are relevant to each team's instructional projects. 

The personal goal for each student within a team should be to acquire knowledge about teaching by applying 
course ideas in actual design practice. The community goal for each team is to contribute to their learning 
communities' (both STEP and TI) growing bank of instructional resources and best-practice cases in educational 
problem solving. Because members of teams are geographically dispersed, it is expected that more and more 
STEP teams will elect to meet and work within the TI (or a similar alternative) environment. 

The Assessment Challenge 

A major aspect of the research reported here involves developing new methods for automated or 
partially-automated assessment of the collaborative work that takes place within on line learning communities. 
There are many reasons why learning community approaches to education require radical new assessment 
models. First, because much work within learning communities is project-oriented and inquiry based rather than 
curriculum driven, the goals of instruction are variable across individuals and groups and are largely determined by 
learners themselves. Since learning outcomes are difficult to specify in advance and are not highly predictable, 
traditional methods of testing are not highly appropriate. A common alternative involves evaluating tangible group 
or individual products that result from group work. For example, members of project teams might submit, for 
evaluation, web sites documenting group projects or portfolios documenting individual development resulting from 
group participation. A third, less-explored alternative involves taking greater advantage of the on-line 
environment's ability to collect data on collaborative process, which is the goal of the work reported here. By 
combining standard AI techniques and frameworks based on current cognitive theories of group interaction, we 
are mining on-line data to obtain evidence for drawing inferences about the quality of on-line work. 

We believe a comprehensive assessment framework for instruction should include evaluation of both tangible and 
intangible cognitive products. However, there are many reasons to develop a process assessment methodology 
as well. One motivation is that on-line learning communities often grow quite large and become too unwieldy for 
supervising teachers to monitor. A reporting system based on process assessment can greatly aid teaching, 
mentoring, and other forms of supervision and network facilitation. For example, process assessment can help 
identify students whose frequencies and patterns of interaction are problematic. Through early identification and 
diagnosis of process deficits, teachers or other types of mentors (both human and machine) can monitor a large 
communication net and direct tailored guidance to specific people and groups in need of attention. 



A Theoretical Framework for Process Assessment of On-Line Interaction 
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We assume that educators, researchers and developers who work with on-line learning communities want to 
foster high-level discourse that supports growth of knowledge within communities, not just social chat. To help 
determine what standards of evidence are appropriate for judging group process on this basis, we developed the 
theoretical framework summarized in Figure 1 . Figure 1 represents a synthesis of four cognitive viewpoints 
regarding the nature of social knowledge construction. We assume that this basic framework can be applied at 
different grain sizes, that is, to small working groups and to larger Internet communities. It is most applicable to 
relatively stable long-term groups that pursue work goals, though some turnover is assumed. The framework is 
less applicable to short-term, ad hoc groups with vague objectives. 

A sociocultural/situative viewpoint f2.31 dominates this framework. From this view, knowledge construction is 
indicated when key "boundary constructs" T41 evolve into mutually understood "new" ideas, behavioral norms, 
and other intangible constructs that shape and are shaped by teamwork. Boundary constructs are ideas brought 
to the problem initially by members of diverse backgrounds, and are so named because they represent shared 
general ideas within overlapping boundaries of disciplinary knowledge. Documentation that boundary constructs 
have emerged, evolved and supported group work is evidence that a knowledge-construction system exists. In 
addition, sociocultural/situative theory implies that the processes of negotiation and apprenticeship drive 
knowledge construction, viewed as boundary construct evolution. Hence, measures reflecting the presence, 
frequency, and quality of these processes within work groups and their larger communities provide evidence for 
their efficacy as knowledge construction entities. 

Figure 1 also represents a sociocognitive perspective, so named because it views knowledge construction as an 
individual cognitive activity that is driven by social interaction. Knowledge construction is seen as involving changes 
and realignments in individuals' mental representations that are associated with their work. Theoretically, as team 
members work together their individual mental models of group tasks, community constructs, and the community 
itself become more aligned with one another. A substantial degree of mental-model alignment is considered a 
necessary precondition for productive work, although some misalignment is desirable F 5,61 . When people with 
different points of view work together, their interactions produce cognitive conflicts that lead to argumentation, 
which in turn drives conceptual change. Hence, from the sociocognitive perspective, insurance that communities 
are operating as knowledge construction systems must be based on evidence of conceptual and belief change 
within individuals, and on evidence that there is a trend toward increasing compatibility among members' 
viewpoints. 

The model also assimilates a literature that sees knowledge construction as both process and product of 
argumentation and informal scientific reasoning. This "critical thinking" [71 viewpoint suggests that negotiation 
processes should be judged against established standards for argument form and content, recognizing that 
argument in conversation is both temporally and socially distributed £8J. It also suggests that the knowledge 
resulting from valid argument is superior to knowledge resulting from less logical processes. Hence, "respect for 
good argument" should be a measurable social norm within knowledge building communities. The degree of 
adherence to critical thinking norms can theoretically be detected and measured in studies of conversation 
patterns and individual beliefs. We believe, however, that valid argument in on-line conversation may be infrequent 
and difficult to detect f91 . Both training and on-line tools may be needed to scaffold good argumentation within 
on-line learning environments. If such tools were available, on-line patterns of use associated with them could help 
us evaluate groups on the basis of their argumentation strategies and their tendency to base judgments and 
decisions on evidence and valid argument. Such tools do not currently exist within TI but could be added. 

Finally, Figure 1 incorporates group information processing theory nO.111 . From this view, assessments of 
groups as knowledge construction entities must consider the processes by which information is shared and 
transformed from one form (e.g., private, unshared knowledge) to another (e.g., a tangible group product). 
Theoretically, successful sharing increases group knowledge and knowledge distribution; hence measures showing 
progressively greater and deeper conceptual alignment among members are valid indicators from the information 
processing viewpoint (as they are from the sociocognitive one). Assessment designs based on the group IP 
perspective can capitalize on research pointing to specific factors known to shape group information processing. 
These include social factors, such as status of individual members. For example, a composite score indicating high 
cultural/professional diversity within groups (indicative of broad knowledge capacity) AND high levels of sharing 
among all social-status categories is one theoretically-sound index for comparing and judging groups as 
knowledge-construction systems. Other factors shaping group information processing include behaviors and 
contexts that impact important information processing phases, such as the attentional phase. For example, one 
useful comparative index is derived from observable signs of group attention deficits. These occur because 
environmental events (e.g., phone calls) frequently distract members from on-line conversations. However, the 
value of this index can be adjusted if there is evidence that members employ technological features of on-line 

O 
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environments to compensate for attentional deficits. For example, members may use an environment's recording 
capabilities to review missed conversation. 

Assessment as Abductive Inference in a Bayesian Net 



Although our theoretical framework is presented as an integrated one, the assessment techniques that follow 
from the four constituent theories differ from one another. For example, the situative model suggests that if 
groups are operating successfully, certain classes of discourse patterns, such as negotiation and apprenticeship, 
must be observed. The measured frequency or strength of such interactions within a prescribed corpus of 
interactive data can help predict the extent to which the group is successful as a knowledge construction system. 
But while the sociocultural/situative view leads to examination of teamwork from observable discourse processes, 
the sociocognitive view leads to protocol, interview and questionnaire techniques that permit researchers and 
teachers to look inside participant's heads— to examine belief change. These data can also be collected on line. 

Assessment thus conceptualized is viewed a problem of abductive inference, or case identification given particular 
theories and relevant evidence pertaining to them. Because case identification is indiscrete and continuous, we 
employ a Bayesian probabilistic framework ri21. The theories predict indicators that can be represented as 
nodes linked by subjective probabilities. Indices and codes representing observable features of on-line interactions 
and tool use provide input to the net. Outputs are propagated probability distributions indicating the degree (e.g., 
strong, medium, weak) to which a target group meets normative standards prescribed by each of the four 
theories. Although these distributions are derived from subjective probabilities and thus cannot be interpreted 
absolutely, they are grounded in theory, constant, and comparable across groups as normative indices. The 
network can thus produce both general assessments and meaningful diagnostic profiles for groups and subgroups 
within large communities. 

To illustrate how this type of network can be used in assessment of online interaction, we present an example 
based on two groups that met regularly in TI. On the day selected for this example, the two groups met at the 
same time (one for around 40 minutes, the other for around 60 minutes) in different virtual rooms to discuss 
different topics related to education. The first 40 minutes of each meeting were analyzed to identify the meeting's 
Group Attention Index, which comprises a branch of the overall network assessing a group's information 
processing ability. 

Table 1 and Table 2 show excerpts taken approximately 15 minutes after the start of each group's meeting. The 
columns differentiate between shared text (seen by all participants), and private text (including whispers and other 
commands seen only by the speaker and any designated recipients). 



In these excerpts, Group A discussion posed fewer attentional demands than Group B . In Group A, while some 
comments were out of sequence (i.e., prompted by a turn not directly above it in the window), all comments 
were both shared and on-task. In contrast, Group B faced multiple potential distractions. The first line shows a 
user apologizing for having technical difficulties. Whispers were exchanged among four participants (NL, NR, CS, 
and BL) on 5 different topics (a previously discussed issue [5:00]; AOL [5:32; 6:34, 7:54]; norms of whispering 
[5:38]; a problem with "J" [6:35, 7:58]; and an out-of-sequence statement [7:45]). Even the whispers 
overlapped, with participants NL, CS, and BL involved in multiple out-of-sequence whispers. As a result, members 
of Group B needed to interpret multiple threads of conversation— some unrelated to the topic being 
discussed— amid technical disruptions and errors. 



The differences between Group A and Group B were quantified using the "Deficits" portion of the probability 
network as shown in figure 2 . which is a sub-branch under the Group Information Processing Attention branch of 
the network. The group attention deficit index (Deficits) was defined as a combination of tool-related deficits 
(Tool_Deficit), group-level distractions (Grp_Distract), and multiple threads in the conversation (AttnToMT). (Note 
that for display purposes, the tool deficits and group distraction sub-branches are shown separately in the 
diagram). 



For each 10-minute interval, we computed per-person averages for involuntary disconnects, early exits, etc., and 
entered the averages as input to the network. Each node in the network shows the probability that the node is in 
a particular state (high/medium/low). The input nodes determined the states of their parent nodes, which in turn 
propagated up to determine the state of the overall attention deficit index. As the Deficits nodes indicate, group A 
had a low degree of attention deficits (64% probability of low deficits; 31% of average deficits; 5% of having high 
deficits). In contrast, group B had higher attention deficits (59% probability of high deficits; 32% probability of 
average deficits, and 9% probability of low deficits). In the larger group attention network (not displayed here) 
these probabilities resulted in group A having 59% probability of being a high-attention group and group B having 
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only a 19% chance of being a high attention group. These results support the differences shown qualitatively in the 
transcript excerpts. 

We are in the process of refining the probability network based on the analysis of additional group meetings. 
Refinements are being made both in the network probabilities (e.g., what combination of values in input nodes 
results in a "high" , "medium", or "low" value for a parent node) and in the thresholds for determining input values 
(e.g., what input qualifies as a "high", "medium", or "low" percentage of whispers per total turns). At this point, 
many indices (such as percentage of errors) can be calculated based on information captured online, while 
others— such as thread count— currently require some human processing. However, the amount of human 
pre-processing will gradually be reduced as coding systems and algorithms for machine processing are developed. 



This research was supported by NSF REPP Grant #CDA-9725528 (Mark Schlager, PI) and a cooperative 
agreement between the National Science Foundation and the University of Wisconsin-Madison, USA (Cooperative 
Agreement No. RED-9452971), which established the National Institute for Science Education. At UW-Madison, 
the National Institute for Science Education is housed in the Wisconsin Center for Education Research and is a 
collaborative effort of the College of Agricultural and Life Sciences, the School of Education, the College of 
Engineering, and the College of Letters and Science. The collaborative effort is also joined by the National Center 
for Improving Science Education, Washington, D.C. Any opinions, findings or conclusions are those of the authors 
and do not necessarily reflect the view of the supporting agencies. 
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6:26 


BL 


And with a little explanation, more were able to see the light. 


6:41 


HC 


think it is important to ask high level questions for all kids 


7:04 


AE 


:agrees with HC 


7:19 


HC 


they surprise you! 


7:30 


RK 


yes they do sometimes 


7:32 


HT 


We are readilng one tf the core books for 4th right now and it is hard 
to get them to go beyond the basics. 


7:32 


BL 


YES!!! Pleasantly so. 


7:38 


AE 


i've seen some classrooms where the teacher actually posts the 
beginning of questions to remember 


7:52 


HC 


:has those in the back of her room 


7:52 


BL 


I would like to try that next year. 



Table 2: Excerpt from Group B's Conversation. Back . 



Time 


Spkr 


Shared text 


Private text 


4:40 


CS 


Sorry folks— -I have AOL. It likes to 
dump me off." 




4:42 


AL 


Time management is a lot like math. 
The more organized and logical I am, 
the smoother my day goes. I waste 
very little time. This allows me relax 
and down time 




4:50 


NL 




Page DJ Thanks... she got it 


5:00 


NR 




(to NL) "thank you 


5:01 


BL 


CS do you have aol loaded at work? 




5:03 


CS 




<who> 


5:04 


NL 


Let's be more specific about that AL 




5:21 


NL 


I notice that you really are organized, 
can you elaborate? 




5:32 


CS 




(to BL) NO, I'm at home..." 


5:38 


NL 




(to BL) Remember to whisper if 
you're off topic, ok? 


6:12 


AL 


I try very hard not to waste steps or 
precious moments. I go through my 
mail over the waste basket. I dump 
what is not needed so I don't have 
these huge piles. I keep files with 
things organized 




6:15 


CS 


I know that I forget anything I don't 
write down..,I even have to E-mail 
myself sometimes. 




6:34 


BL 




(to NL) You mean AOL isn't a 
legitmate use of time mangenent 
?<w> 


6:35 


NL 




(to BL) Did we get the situation 
with J worked out? 


6:39 


NL 


:nods 




6:41 


CS 


SO AL— it sounds like those little time 
savers you do add up to big time 
savings overall... 




6:59 


CS 


OK— How about saving time with 
this: 




7:01 


AL 




<error> 
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7:05 


NL 


NR, you're an old timer at this school 
teaching business... what ideas do 
you have? 




7:06 


cs 


(projects prepared text) 




7:10 


BL 




(to NL) He looks pouty but he 
hasn't complained :) 


7:45 


cs 




(to NL) "Didn't mean to step on 
your toes..." 


7:54 


BL 




(to CS) have you asked about 
loading aol at school? 


7:58 


NL 




(to BL) I think it's totally your call 
unless he wants to complain to 
higher authorities 
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